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METHOD AND APPARATUS FOR VIDE O DECODING ON A 
MULTIPROCESSOR SYSTEM 



ETELD OF THE INVENTION 

The present invention relates computers and multimedia 
processing, and, in particular, to the decoding compressed video on a 
multiprocessor system. 

BACKGROUND OF THE INVENTION 

Recent advances in computer technology have placed video 
applications within the reach of more common applications. For example, 
high Definition Television, Boradcast satellite Service, Cable TV 
distribution on optical networks, Electron Cinema, Interactive Storage 
Media, Multimedia Mailing, Networked Data base Services, corporate 
Internet training and conferencing, Remote Video Surveillance and others 
are now becoming practical video applications. 

The large amounts of data needed to make video available in all 
these cases has lead to the adoption of Moving Picture Experts Group 1 
(MPEG) and MPEG-2 standards for motion video compression and 
decompression. These standards significantly reduce bandwidth and 
storage space requirements. As a result, MPEG-1 and MPEG-2 are used in 
many video applications and are continuing to become more popular. 
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5 However, one feature limiting widespread use of the MPEG 

standard is it computational complexity. Video encoding and decoding 
under MPEG is expensive, often too costly for single processors to achieve 
real-time performance in software for displays of acceptable resolution 
and size. The computational demands grow as users desire higher quality 

10 video. 

Encoding is more expensive than decoding, but can typically be 
done offline and, therefore, may not need to be done in real-time. 
Decoding, however, typically requires real-time performance. 
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5 SUMMARY OF THE INVENTION 



The present invention provides a method and apparatus for 
decoding compressed video. The method includes reading a stream of 

10 compressed video into a memory. The video includes multiple pictures, 
with each picture having one or more independent elements. Thereafter, 
assigning, via a first processor of a group of processors sharing the 
memory, at least one independent element per processor to be decoded by 
the processors in parallel; and decoding the independent elements of the 

15 video stream in parallel. 
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5 BRIEF DESCRIPTION OF THE DRAWINGS 



Figure 1 illustrates the hierarchy of layers in an MPEG bitstream. 

Figure 2 illustrates a block diagram of encoding and decoding 
10 video data, according to one embodiment. 

Figure 3 illustrates a block diagram of a picture slice structure. 

Figure 4 illustrates a flow diagram describing the steps of decoding 
15 video data with multiple processors in parallel, according to one 
embodiment. 

Figure 5 illustrates a multiprocessor computer system having a 
computer readable medium with instructions stored thereon according to 
20 one embodiment. 
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5 DETAILED DESCRIPTION 

A method and apparatus for decoding compressed video on a 
multiprocessor system is disclosed. In the following description, for 
purposes of explanation, specific nomenclature is set forth to provide a 
thorough understanding of the present invention. However, it will be 

10 apparent to one skilled in the art that these specific details are not 

required in order to practice the present invention. For example, although 
the embodiments discussed below describe the present invention being 
used to decode video data encoded in accordance with the MPEG 
standard, alternative video encoding/ decoding standards could be used 

15 without departing from the present invention. 

MPEG Decoding 

In its general form, an MPEG system stream is made up of two 
20 layers: the system layer contains timing and other information needed to 
demultiplex the audio and video streams and to synchronize audio and 
video during playback; and, the compression layer which includes the 
audio and video data streams. 

One embodiment of the hierarchy of layers in an MPEG bitstream 
25 is arranged in the following order, as illustrated in Figure 1: Sequence, 
Group of Pictures (GOP), Picture Slice, Macroblock, and Block. The 
different parts of the stream are marked with unique, byte-aligned codes 
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5 called startcodes. These startcodes are used both to identify certain parts 
of the stream and to allow random access into the video stream. 



The highest level in the layering is the sequence level A sequence 
is made up of groups of pictures (GOPs), Each GOP is a grouping of a 
10 number of adjacent pictures. One purpose in creating such an identifiable 
grouping is to provide a point of random access into the video stream for 
play control functions (fast forward, reverse, etc.). 

Within each GOP are a number of pictures. In MPEG-2 interlaced 
15 video is supported so each picture corresponds to either a frame (for 
progressive or interlaced video streams) or a field (for interlaced video 
streams) in the original stream. 

Pictures are further subdivided into slices, each of which defines a 
20 fragment of a row in the picture. Slices comprise a series of macroblocks, 
which in one embodiment are 16x16 pixel groups containing the 
luminance and chrominance data for those pixels in the decoded picture. 

Macroblocks are divided into blocks. A block in one embodiment 
25 is an 8x8-pixel group that further describes the luminance or chrominance 
for that group of pixels. Blocks are the basic unit of data at which the 
decoder processes the encoded video stream. Macroblocks and blocks do 
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5 not have startcodes associated with them; their boundaries are discovered 
implicitly while decoding. 

Encoding 

As described above, the block is the basic unit of data processing. 
10 As illustrated in Figure 2, in one embodiment for each block of data in the 
video sequence, the encoder typically performs following five steps to 
produce an encoded block: motion estimation, discrete cosine transform 
(DCT), quantization, and run-length and Huffman coding. 

15 In the first stage, motion estimation, the encoder tries to take 

advantage of temporal redundancies among pictures. The next four 
stages of the encoder take advantage of spatial correlation in compressing 
the video sequence. The result of performing these five encoding stages 
on all blocks in a video-sequence is an MPEG encoded video stream. A 

20 stream may be encoded once and then transmitted across a transmission 
media and/or stored as needed. However, decoding is typically 
necessary each time the stream is viewed. 

Decoding 

25 The decoding process for an MPEG encoded stream typically 

performs the five encoding stages in reverse order. First Huffman and 
run-length decoding are applied to generate the quantized block. Then 
inverse quantization is performed to obtain the block's frequency 
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5 spectrum. From this, the inverse discrete cosine transform is taken. Then, 
if necessary, motion compensation may be used to generate the final 
macroblock of data from the prediction error, motion vectors, and 
reference picture. 

10 A significant difference between the encoding and decoding 

processes is that in the former, much time is spent in motion estimation as 
it is necessary to search for the most similar macroblock in the reference 
picture(s), whereas in the latter, motion vectors are already available, 
which makes motion compensation much cheaper than motion estimation. 

15 

Decoding Video Data with Multiple Processors in Parallel 

In order to increase the speed of decoding MPEG video data to 
provide the video data for more applications, one embodiment provides 

20 parallel decoding of the video data among multiple processors in a single 
system, sharing a memory. In particular, in one embodiment, the 
workload of decoding the video data is distributed among the set of 
processors at the slice level. Li alternative embodiments, the workload of 
decoding the video data could be distributed among the set of processors 

25 at a level other than the slice level, without departing from the present 
invention. 
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5 In particular, as defined by the standard, a slice in MPEG is a series 

of an arbitrary number of macroblocks within one row of the picture. 
Figure 3 illustrates a general slice structure. Each slice contains at least 
one macroblock, and consecutive slices may not overlap. Slices occur in 
the bitstream in the order in which they are encountered, starting at the 

10 upper-left of the picture and proceeding by raster-scan order from left to 
right and top to bottom. However, slices do not need to maintain the 
same structure from picture to picture: there may be different numbers of 
slices and/or different slice sizes. 

15 Figure 4 is a flow diagram describing the steps of decoding the 

video data among multiple processors in parallel, according to one 
embodiment. In step 402 a first processor executing a first process, 
scans/reads a video sequence into memory from a disk or some other 
source of compressed video. 

20 

In step 404, the first processor further reads the video sequence to 
identify slice startcodes to identify the beginning of slices within the video 
sequence. In step 406, the first processor further reads the macroblocks 
within each slice to estimate the workload associated with decoding a 
25 respective slice. 



In step 408, the first processor assigns one or more slices to be 
decoded, to each processor within a group of processors included in a 
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5 system. In one embodiment, the slices are assigned to the different 
processors in a manner to spread the workload evenly among the 
processors. For example, if a first slice includes many macroblocks, it 
presumably will take longer to decode. Therefore, a second processor 
would be assigned fewer slices to decode because at least one slice 
10 presumably will take longer to decode. Subsequent adjacent slices may 
include significantly less macroblocks. Therefore, a third processor, may 
be assigned more slices to decode, with each slice containing less 
macroblocks. In the end, each of the processors, in one embodiment, is to 
receive a comparable workload of slices to decode. 

15 

In step 410, the first processor writes in a shared memory, the first 
and last assigned slices as local variables for each of the processors that 
are provided to decode the slices. For example, if the second processor is 
to decode slices 1-3, a third processor to decode slices 4-8, and a fourth 

20 processor decode slices 9-12, the slice numbers are written in memory as 
local variables for each of the respective processors. One benefit to 
assigning the slices by identifying the slices as local variables for the 
respective processors, as opposed to providing the slices in a jointly 
accessible task/ slice queue, is the avoidance of conflicts between 

25 processors in accessing the queue to identify slices to be decoded. 



For example, in the case of using a slice queue, once a processor is 
done decoding a slice it would return to the queue to determine the next 
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5 slice to be decoded. In addition, the processor would also increment a 
pointer identifying the next slice to be decoded so that the next time a 
processor accesses the queue it will receive the subsequent slice to be 
decoded. Moreover, the queue would be accessed fairly frequent. As a 
result, conflicts between the processors accessing the queue would also be 
1 0 frequent. However, by assigning the slices to be decoded as local 
variables for each of the respective processors, conflicts between the 
processors in accessing the memory is significantly decreased. 



In step 412, the respective processors read their respective local 
15 variables to determine which slices are to be decoded, and thereafter read 
the slices from memory and decode the slices in accordance with the 
decoding procedures discussed above. In one embodiment, the processor 
that assigns the slices to be decoded also assigns to it's self, one or more 
slices to be decoded, which the processor thereafter proceeds to decode 
20 along with the other processors. Moreover, in one embodiment, the 
respective processors perform the steps of decoding the slices by 
executing software routines stored in memory, provided for performing 
the decoding steps as discussed above (e.g., Huffman decoding, Rim 
length decoding, etc). 

25 

Thereafter, the first processor continues to scan/read in the video 
data and assign contiguous slices to the processors for decoding, for the 
desired quantity of video that is to be decoded. Furthermore, the method 
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5 of decoding the video data by distributing the workload among multiple 
processors, as described above, can be performed in real-time. 

Figure 5 illustrates a system that is capable of decoding multiple 
slices in parallel with multiple processors, according to one embodiment. 
10 As illustrated, the system includes multiple processors sharing a memory 
(e.g., Symmetric Multiple Processor architecture). In an alternative 
embodiment, the multiple processors could access separate memories, 
rather than sharing the same memory, without departing from the present 
invention. 

15 

Moreover, as shown in the memory of the system in Figure 5, the 
unit of logic to perform the method of decoding the video data through 
the use of multiple processors in parallel, as described above, could be 
provided a set of computer instructions to be executed by one or more of 

20 the processors. In addition, the logic to perform the steps of decoding the 
video data could also be provided /stored in memory as a set of computer 
instructions (e.g. MPEG decoder). The instructions to perform the 
methods as described above could alternatively be stored on other forms 
of computer/ machine readable medium, including magnetic and optical 

25 disks. For example, method of the present invention can be stored on 
computer/ machine readable mediums, such as magnetic disks or optical 
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5 disks, that are accessible via a disk drive (or computer-readable medium 
drive), such as the disk drive shown in Figure 5. 

Alternatively, the logic to perform the methods as discussed above, 
including the method of decoding video data via multiple processors in 
parallel, could be implemented in additional computer and/ or machine 

10 readable mediums, such as discrete hardware components such as large- 
scale integrated circuits (LSI's), application-specific integrated circuits 
(ASIC's), firmware such as electrically erasable programmable read-only 
memory (EEPROM's); and, electrical, optical, acoustical or other forms of 
propagated signals (e.g., carrier waves, infrared signals, digital signals, 

15 etc.). 
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CLAIMS 

What is claimed is: 

1 1. A method for decoding compressed video comprising: 

2 reading a stream of compressed video into a memory, said 

3 video having multiple pictures, each picture having one or more 

4 independent elements; 

5 assigning, via a first processor of a group of processors sharing 

6 said memory, at least one independent element per processor to be 

7 decoded by the processors in parallel; and 

8 decoding the independent elements of the video in parallel. 

1 2. The method of claim 1, wherein the independent elements 

2 include slices. 

1 3. The method of claim 2, wherein assigning the independent 

2 elements includes assigning a varying number of slices to individual 

3 processors. 

1 4. The method of claim 3, wherein assigning the independent 

2 elements includes assigning a comparable work load to the processors. 

1 5. The method of claim 4, wherein assigning the independent 

2 elements includes placing in memory as a local variable, for each 

3 processor, the slices to be decoded by a respective processor. 
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1 6. The method of claim 5, wherein each slice includes at least one 

2 macroblock. 

1 7. The method of claim 6, wherein said video is encoded in MPEG. 

1 8. The method of claim 7, wherein the method of decoding is 

2 performed in real-time. 

1 9. A computer-readable medium having stored thereon a set of 

2 instructions, said set of instruction for decoding compressed video, which 

3 when executed by a processor, cause said processor to perform a method 

4 comprising: 

5 reading a stream of compressed video into a memory, said 

6 video having multiple pictures, each picture having one or more 

7 independent elements; 

8 assigning, via a first processor of a group of processors sharing 

9 said memory, at least one independent element per processor to be 

10 decoded by the processors in parallel; and 

1 1 decoding the independent elements of the video in parallel. 

1 10. The computer-readable medium of claim 9, wherein the 

2 independent elements include slices. 



Express Mail No.: EL431890855US 



15 



042390.P7940 



1 11. The computer-readable medium of claim 10, wherein assigning 

2 the independent elements includes assigning a varying number of slices to 

3 individual processors. 

1 12. The computer-readable medium of claim 11, wherein assigning 

2 the independent elements includes assigning a comparable work load to 

3 the processors. 

1 13. The computer-readable medium of claim 12, wherein assigning 

2 the independent elements includes placing in memory as a local variable, 

3 for each processor, the slices to be decoded by a respective processor. 

1 14. The computer-readable medium of claim 13, wherein each slice 

2 includes at least one macroblock. 

1 15. The computer-readable medium of claim 14, wherein said 

2 video is encoded in MPEG standard. 

1 16. The computer-readable medium of claim 15, wherein the 

2 method of decoding is performed in real-time. 

1 17. A computer system comprising: 

2 a plurality of processors ; 

3 a memory coupled to said plurality of processors; 
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4 a first unit of logic to read a stream of compressed video into 

5 said memory, said video having multiple pictures, with each picture 

6 having one or more independent elements; and 

7 said first unit of logic further assigns, via a first processor of 

8 said group of processors sharing said memory, at least one independent 

9 element per processor to be decoded by the processors in parallel. 

1 18. The computer system of claim 17, wherein the independent 

2 elements include slices. 

1 19. The computer system of claim 18, wherein said first unit of 

2 logic assigns a varying number of slices to individual processors. 

1 20. The computer system of claim 19, wherein said first unit of 

2 logic assigns a comparable work load to the processors. 

1 21. The computer system of claim 20, wherein said first unit of 

2 logic places in memory as a local variable, for each processor, the slices to 

3 be decoded by a respective processor. 

1 22. The computer system of claim 21, wherein each slice includes at 

2 least one macroblock. 
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1 23. The computer system of claim 22, wherein said video is 

2 encoded in MPEG standard. 

1 24. The computer system of claim 23, wherein system computer 

2 system decodes said video in real-time. 



Express Mail No.: EL431890855US 



18 



042390.P7940 



ABSTRACT OF THE DISCLOSURE 



A method and apparatus for decoding compressed video. The 
method includes reading a stream of compressed video into a memory. 
5 The video includes multiple pictures, with each picture having one or 

more independent elements. Thereafter, assigning, via a first processor of 
a group of processors sharing the memory, at least one independent 
element per processor to be decoded by the processors in parallel; and 
decoding the independent elements of the video stream in parallel. 
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Anthony DeSanctis, Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; Robert Andrew Diehl, Reg. No. 
40,992; Matthew C. Fagan, Reg. No. 37,542; Tarek N. Fahmi, Reg. No. 41,402; James Y. Go, Reg. No. 
40,621 ; James A. Henry, Reg. No. 41,064; Willmore F. Holbrow III, Reg. No. P41,845; Sheryl Sue 
Holloway, Reg. No. 37,850; George W Hoover II, Reg. No. 32,992; Eric S. Hyman, Reg. No. 30,139; Dag 
H. Johansen, Reg. No. 36,172; William W. Kidd, Reg. No. 31,772; Erica W. Kuo, Reg. No. 42,775; Michael 
J. Mallie, Reg. No. 36,591; Andre L Marais, under 37 C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. 
42,879; Darren J. Milliken, Reg. 42,004; Lisa A. Norris, Reg. No. P44,976; Chun M. Ng, Reg. No. 36,878; 
Thien T. Nguyen, Reg. No. 43,835; Thinh V. Nguyen, Reg. No. 42,034; Dennis A. Nicholls, Reg. No. 
42,036; Kimberley G. Nobles, Reg. No. 38,255; Daniel E. Ovanezian, Reg. No. 41,236; Babak Redjaian, 
Reg. No. 42,096; William F. Ryann, Reg. 44,313; James H. Salter, Reg. No. 35,668; William W. Schaal, 
Reg. No. 39,018; James C. Scheller, Reg. No. 31,195; Jeffrey Sam Smith, Reg. No. 39,377; Maria 
McCormack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff, Reg. No. 25,128; Judith A. Szepesi, Reg. No. 
39,393; Vincent P. Tassinari, Reg. No. 42,179; Edwin H. Taylor, Reg. No. 25,129; John F. Travis, Reg. 
No. 43,203; George G. C. Tseng, Reg. No. 41,355; Joseph A. Twarowski, Reg. No. 42,191; Lester J. 
Vincent, Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 41,364; John Patrick Ward, Reg. No. 40,216; 
Charles T. J. Weigell, Reg. No. 43,398; Kirk D. Williams, Reg. No. 42,229; James M. Wu, Reg. No. 
P45,241; Steven D. Yates, Reg. No. 42,242; Ben J. Yorks, Reg. No. 33,609; and Norman Zafman, Reg. 
No. 26,250; my patent attorneys, and Andrew C. Chen, Reg. No. 43,544; Justin M. Dillon, Reg. No. 
42,486; Paramita Ghosh, Reg. No. 42,806; and Sang Hui Kim, Reg. No. 40,450; my patent agents, of 
BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th 
Floor, Los Angeles, California 90025, telephone (310) 207-3800, and Alan K. Aldous, Reg. No. 31,905; 
Robert D. Anderson, Reg. No. 33,826; Joseph R. Bond, Reg. No. 36,458; Richard C. Calderwood, Reg. 
No. 35,468; Jeffrey S. Draeger, Reg. No. 41 ,000; Cynthia Thomas Faatz, Reg No. 39,973; Sean 
Fitzgerald, Reg. No. 32,027; Seth Z. Kalson, Reg. No. 40,670; David J. Kaplan, Reg. No. 41,105; Charles 
A. Mirho, Reg. No. 41,199; Leo V. Novakoski, Reg. No. 37,198; Naomi Obinata, Reg. No. 39,320; 
Thomas C. Reynolds, Reg. No. 32,488; Kenneth M. Seddon, Reg. No. 43,105; Mark Seeley, Reg. No. 
32,299; Steven P. Skabrat, Reg. No. 36,279; Howard A. Skaist, Reg. No. 36,008; Steven C. Stewart, Reg. 
No. 33,555; Raymond J. Werner, Reg. No. 34,752; Robert G. Winkle, Reg. No. 37,474; and Charles K. 
Young, Reg. No. 39,435; my patent attorneys, and Thomas Raleigh Lane, Reg. No. 42,781; Calvin E. 
Wells; Reg. No. P43,256, Peter Lam, Reg. No. P44,855; and Gene I. Su, Reg. No. 45,140; my patent 
agents, of INTEL CORPORATION; and James R. Thein, Reg. No. 31,710, my patent attorney; with full 
power of substitution and revocation, to prosecute this application and to transact all business in the 
Patent and Trademark Office connected herewith. 



APPENDIX B 



Title 37, Code of Federal Regulations, Section 1 .56 
Duty to Disclose Information Material to Patentability 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that individual 
to be material to patentability as defined in this section. The duty to disclosure information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit information which is not material 
to the patentability of any existing claim. The duty to disclosure all information known to be material to 
patentability is deemed to be satisfied if all information known to be material to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1.98. However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 



-5- 



